Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry

Deep learning has achieved spectacular performance in image and speech recognition and synthesis. It outperforms other machine learning algorithms in problems where large amounts of data are available. In the area of measurement technology, instruments based on the photonic time stretch have established record real-time measurement throughput in spectroscopy, optical coherence tomography, and imaging flow cytometry. These extreme-throughput instruments generate approximately 1 Tbit/s of continuous measurement data and have led to the discovery of rare phenomena in nonlinear and complex systems as well as new types of biomedical instruments. Owing to the abundance of data they generate, time-stretch instruments are a natural fit to deep learning classification. Previously we had shown that high-throughput label-free cell classification with high accuracy can be achieved through a combination of time-stretch microscopy, image processing and feature extraction, followed by deep learning for finding cancer cells in the blood. Such a technology holds promise for early detection of primary cancer or metastasis. Here we describe a new deep learning pipeline, which entirely avoids the slow and computationally costly signal processing and feature extraction steps by a convolutional neural network that directly operates on the measured signals. The improvement in computational efficiency enables low-latency inference and makes this pipeline suitable for cell sorting via deep learning. Our neural network takes less than a few milliseconds to classify the cells, fast enough to provide a decision to a cell sorter for real-time separation of individual target cells. We demonstrate the applicability of our new method in the classification of OT-II white blood cells and SW-480 epithelial cancer cells with more than 95% accuracy in a label-free fashion

High-Speed Imaging and Optical Sensing Systems for Biomedical Applications

High-throughput real-time optical sensing and imaging instruments for capture and analysis of fast phenomena are among the most essential tools for scientific, industrial, military, and most importantly biomedical applications. The key challenge in these instruments is the fundamental trade-off between speed and sensitivity of the measurement system due to the limited signal energy collected in each measurement window. Based on two enabling technologies, namely photonic time-stretch dispersive Fourier transform and optical amplification, we developed several novel high-throughput optical measurement tools for applications such as flow cytometry, vibrometry, and volumetric scanning.We demonstrated optical Raman amplification at about 800 nm wavelength for the first time and extended time-stretch dispersive Fourier transform to this region of electromagnetic spectrum. We used this enabling technology to make an ultrafast three-dimensional laser scanner with about hundred thousand scans per second and an imaging vibrometer with nanometer-scale axial resolution. We also employed our high-speed laser scanner to perform label-free cell screening in flow. One of the fundamental challenges in cell analysis is the undesirable impact of cell labeling on cellular behavior. To eliminate the need for these labels, while keeping the cell classification accuracy high, additional label-free parameters such as precise measurement of the cell protein concentration is required. We introduced a high-accuracy label-free imaging flow cytometer based on simultaneous measurement of morphology and optical path length through the cell at flow speeds as high as a few meters per second. Finally, the ultimate challenge in ultra-high-throughput instrumentation is the storage and analysis of the torrent of generated data. As an example, our imaging flow cytometer generates about ten terabytes of cell images over a course of one hour acquisition, which captures images of every single cell in more than two milliliters of sample e.g. blood. We enabled practical use of these big data volumes by efficient combination of analog preprocessing techniques such as quadrature demodulation with parallel storage and digital post-processing

Designing an Efficient End-to-end Machine Learning Pipeline for Real-time Empty-shelf Detection

On-Shelf Availability (OSA) of products in retail stores is a critical business criterion in the fast moving consumer goods and retails sector. When a product is out-of-stock (OOS) and a customer cannot find it on its designed shelf, this motivates the customer to store-switching or buying nothing, which causes fall in future sales and demands. Retailers are employing several approaches to detect empty shelves and ensure high OSA of products; however, such methods are generally ineffective and infeasible since they are either manual, expensive or less accurate. Recently machine learning based solutions have been proposed, but they suffer from high computational cost and low accuracy problem due to lack of large annotated datasets of on-shelf products. Here, we present an elegant approach for designing an end-to-end machine learning (ML) pipeline for real-time empty shelf detection. Considering the strong dependency between the quality of ML models and the quality of data, we focus on the importance of proper data collection, cleaning and correct data annotation before delving into modeling. Since an empty-shelf detection solution should be computationally-efficient for real-time predictions, we explore different run-time optimizations to improve the model performance. Our dataset contains 1000 images, collected and annotated by following well-defined guidelines. Our low-latency model achieves a mean average F1-score of 68.5%, and can process up to 67 images/s on Intel Xeon Gold and up to 860 images/s on an A100 GPU.Comment: 7 figures, 3 tables, 10 page

Artificial intelligence in label-free microscopy: biological cell classification by time stretch

This book introduces time-stretch quantitative phase imaging (TS-QPI), a high-throughput label-free imaging flow cytometer developed for big data acquisition and analysis in phenotypic screening. TS-QPI is able to capture quantitative optical phase and intensity images simultaneously, enabling high-content cell analysis, cancer diagnostics, personalized genomics, and drug development. The authors also demonstrate a complete machine learning pipeline that performs optical phase measurement, image processing, feature extraction, and classification, enabling high-throughput quantitative imaging that achieves record high accuracy in label -free cellular phenotypic screening and opens up a new path to data-driven diagnosis. • Demonstrates how machine learning is used in high-speed microscopy imaging to facilitate medical diagnosis; • Provides a systematic and comprehensive illustration of time stretch technology; • Enables multidisciplinary application, including industrial, biomedical, and artificial intelligence